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ABSTRACT 

Large-scale  team  training  presents  a  challenge  for  instructors  who  must  coordinate  after  action  review  for 
distributed  teams.  This  paper  presents  a  visual  timeline-based  debrief  toolset  that  enables  instructors  to  quickly 
construct  and  present  playback  vignettes  for  salient  training  points. 

The  AAIRS  (After  Action  Intelligent  Review  System)  application  being  developed  for  the  Marine  Corps’ 
CACCTUS  (Combined  Arms  Command  and  Control  Trainer  Upgrade  System)  identifies  training  points  through  the 
use  of  intelligent  data  collection  and  causal  analysis  methods.  AAIRS  records  the  training  mission  execution,  which 
includes  a  synchronized  collection  of  exercising  force  audio  communications  and  human  in  the  loop  interaction  with 
system  components  (operational  C4I  and  simulator  interface  tools).  Key  exercise  data  relevant  to  training  points  is 
tabulated  by  the  debrief  construction  tool  and  presented  visually  along  a  timeline  for  instructor  review.  The  time 
consuming  process  of  reviewing  sequential  recorded  radio  communications  is  eased  by  visually  representing 
individual  transmissions  on  the  timeline,  grouped  into  dialogs  and  annotated  with  speech  analysis  results.  The 
tool’s  presentation  capability  allows  instructors  to  preview  their  planned  debrief  presentation  and  customize  it  prior 
to  formal  debrief.  This  facilitates  a  highly  configurable  after  action  review,  where  the  presentation  of  each  training 
point  in  the  debrief  can  include  descriptive  causal  analysis  text,  voice  communications  playback,  and  3D  views  of 
the  battlefield,  either  frozen  in  time  (visually,  a  “snapshot”)  or  played  back  as  vignettes. 
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INTRODUCTION 

Large-scale  team  training  presents  a  challenge  for 
instructors  who  must  coordinate  after  action  review  for 
distributed  teams.  This  paper  presents  the  concept  for 
and  description  of  a  visually-presented,  and  timeline- 
based,  debrief  toolset  that  enables  instructors  to 
quickly  identify,  construct,  and  present  playback 
vignettes  for  salient  training  points. 

The  AAIRS  (After  Action  Intelligent  Review  System) 
application  being  developed  for  the  Marine  Corps’ 
CACCTUS  (Combined  Arms  Command  and  Control 
Trainer  Upgrade  System)  identifies  training  points 
through  the  use  of  intelligent  data  collection  and  causal 
analysis  methods.  AAIRS  records  the  training  mission 
execution,  which  includes  a  synchronized  collection  of 
exercising  force  radio  communications  and  human  in 
the  loop  interaction  with  system  components 
(operational  C4I  and  simulation/simulator  interface 
tools).  Key  exercise  data  relevant  to  training  points  are 
tabulated  by  the  debrief  construction  tool  and 
presented  in  a  visual  format  along  a  timeline  for 
instructor  review.  The  time  consuming  process  of 
reviewing  sequential  recorded  radio  communications  is 
eased  by  visually  representing  individual  transmissions 
on  the  timeline,  grouped  into  dialogs  and  annotated 
with  speech  analysis  results.  The  tool’s  presentation 
capability  allows  instructors  to  preview  their  planned 
debrief  presentation  and  customize  it  prior  to  formal 
debrief.  This  facilitates  a  highly  configurable  after 
action  review,  where  the  presentation  of  each  training 
point  in  the  debrief  can  include  descriptive  causal 
analysis  text,  radio  communications  playback,  and  3D 
views  of  the  battlefield,  either  frozen  in  time  (a  visual 
“snapshot”)  or  played  back  as  vignettes. 

This  paper  describes  the  training  requirements  that 
constitute  the  objectives  for  the  subject  effort,  with  a 
short  discussion  of  several  previous  efforts  that  have 
provided  timeline-based  debrief  playback  capabilities. 
Initial  feedback  from  the  user  community  is  also 
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discussed,  as  well  as  the  way  forward  for  continued 
concept  development. 


TRAINING  OBJECTIVES 

CACCTUS  will  support  a  live,  virtual,  and 
constructive  training  environment  that  facilitates  the 
interaction  between  and  among  levels  and  echelons  of 
command  and  control  agencies  normally  found  in  the 
tactical  environment  in  the  conduct  of  real-time 
combined  arms  fire  support  operations.  Combined 
arms  exercises  train  and  rehearse  the  exercising  force 
(EXFOR)  personnel  in  the  tasks  of  coordinating 
multiple  supporting  arms  with  maneuver.  Training 
exercises  take  place  in  Combined  Arms  Staff  Trainer 
(CAST)  facilities,  and  may  involve  100  or  more 
participants  at  various  stations  in  the  facility,  carrying 
out  their  respective  operational  responsibilities.  There 
is  an  emphasis  on  providing  the  EXFOR  the 
experience  of  performing  a  role  in  scenario-based 
combined  arms  exercises,  so  that  training  is 
experiential.  As  a  result,  EXFOR  responsibilities 
during  training  mirror  those  during  operational  actions, 
including  the  employment  of  skills  in  communication 
and  coordination,  and  tactics,  techniques,  and 
procedures  in  support  of  specific  training  goals. 

As  an  example,  the  battalion  level  Fire  Support 
Coordinator  (FSC)  is  responsible  for  clearing  requests 
for  fire  missions  and  air  strikes  from  subordinate  units, 
and  therefore  must  maintain  a  clear  and  accurate 
operational  picture.  This  also  requires  coordination 
with  senior  and  adjacent  units,  with  a  constant  flow  of 
requested  and  disseminated  information.  In  a  specific 
example,  if  a  forward  observer  requests  an  artillery  fire 
mission,  the  FSC  must  have  a  correct  operational 
picture  in  order  to  make  a  determination  that  is  both 
timely  and  consistent  with  the  scheme  of  maneuver, 
rules  of  engagement,  and  safety  constraints. 
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Within  this  operational  context,  the  training  objective 
is  not  only  to  detect  execution  event  outcomes  and 
report  key  training  points  back  to  EXFOR,  but  also  to 
trace  the  decision-making  and  possible  errors  that 
precipitated  the  event  outcomes.  This  end-to-end 
objective  requires  a  robust  data  collection  and  analysis 
capability  that  monitors  events  and  actions  during  an 
exercise  and  stores  data  in  a  form  that  can  be  rapidly 
retrieved  in  after  action.  From  a  user  perspective,  a 
debrief  toolset  is  required  that  supports  a  two-step 
process.  First,  human  instructors  must  have  Debrief 
Construction  tools  with  which  they  can  quickly  form 
their  own  knowledge  of  the  training  points,  and 
assemble  debrief  materials  accordingly.  Second, 
Debrief  Presentation  tools  must  provide  clear  context 
to  the  training  audience,  ideally  in  a  visual  format  that 
doesn’t  obscure  training  points  in  text  and  statistics. 

The  need  for  such  capabilities  is  common  to  many 
team  training  domains,  especially  where 
communications  and  coordination  play  a  key  role  in 
success,  and  thus  where  the  after  action  aims  to 
highlight  specific  decision  points  in  playback  and  show 
their  relationships  to  simulation  events.  Accordingly, 
this  paper  focuses  less  on  the  approaches  for 
supporting  specific  domain  training  points  (e.g., 
battlespace  geometry  conflicts  in  combined  arms 
operations),  and  more  on  the  general  methodologies  for 
providing  flexible  tools  for  quick  exercise  review,  and 
debrief  preparation  and  presentation.  The  intention  is 
to  support  objectives  ranging  from  a  quick 
automatically  generated  debrief,  to  situations  where 
instructors,  operators,  or  controllers  seek  to  review 
training  points  in  detail  before  presenting  a  debrief, 
and  to  possibly  customize  the  debrief  in  a  variety  of 
ways,  using  tools  to  make  this  process  fast  and  clear. 

Debrief  Construction  Objectives 

The  simplest  Debrief  Construction  use  case  involves 
assembling  the  automatically  generated  debrief 
materials  in  one  step,  if  the  instructor  is  satisfied  that 
no  customization  is  required.  In  cases  where 
instructors  prefer  to  review  training  points  in  detail  and 
tailor  them  for  debrief,  the  emphasis  on  visual 
presentation  of  exercise  data  accelerates  these  tasks, 
with  a  goal  of  keeping  total  preparation  time  to  30 
minutes  or  less.  The  specific  objectives  of  the  Debrief 
Construction  tools  include: 

•  Make  use  of  automated  analysis  and  event 
detection  capabilities  performed  during 
exercise  execution  to  collect  training  points 
for  after  action  review. 


•  Provide  sorting  and  filtering  capabilities  to 
support  the  review  of  all  training  points 
generated  during  the  exercise. 

•  Enable  the  creation  of  custom  training  points, 
when  desired.  For  example,  an  instructor  may 
want  to  draw  attention  to  conditions  of 
interest  such  as  cases  of  good  execution  or 
decision-making  that  are  not  automatically 
identified  by  the  system. 

•  Represent  exercise  events  along  a  visual 
timeline,  in  a  symbolic  format  consistent  with 
the  operational  domain.  The  primary  purpose 
is  to  make  the  temporal  relationships  between 
actions  immediately  apparent.  For  example, 
was  an  indirect  fire  mission  approved  before 
or  after  a  friendly  force  ground  unit  initiated  a 
movement? 

•  Provide  a  visual  representation  for  voice 
communications  and  dialogs  in  the  timeline 
structure  in  order  to  give  faster  access  to 
transmissions  of  interest  compared  to  linear 
audio  playback.  Automatically  annotate 
transmissions  with  speech  recognition  results 
to  indicate  content. 

•  Automatically  configure  an  initial  debrief 
playback  for  each  training  point,  consisting  of 
synchronized  audio  communications  with  a 
3D  view  (and  optional  2D  view)  of 
movements  and  missions  executed  on  the 
battlefield. 

•  Provide  tools  to  preview  and  customize  the 
initially  prepared  debrief  content  for  any 
given  training  point. 

Debrief  Presentation  Objectives 

The  objectives  for  Debrief  Presentation  are 
fundamentally  similar  to  the  goals  of  the  preview 
capabilities  in  Debrief  Construction.  Both  require  a 
visual  platform  for  quickly  conveying  what  happened 
and  what  the  training  point  is,  with  respect  to  exercise 
events.  Further,  it  is  critical  to  provide  instructors  with 
a  common  look  and  feel  in  the  tools  they  use  for  both 
functions,  as  they  facilitate  both  tasks.  The  Debrief 
Presentation  objectives  include: 

•  Provide  visual  playback  of  events  from 
exercise  execution,  synchronized  with  audio 
playback  of  voice  communications. 
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•  Include  the  event  timeline  with  playback,  in 
order  to  maintain  the  context  with  operational 
events  outside  the  visualization  field  of  view 
or  moment  in  time. 

•  Provide  playback  controls  to  freeze  or  jump 
ahead  in  time. 

•  Provide  the  ability  to  change  the  3D  point  of 
view  during  playback. 

•  Provide  text  display  capabilities,  which  can  be 
called  up  as  overlays  during  playback,  to 
present  automatically  generated  causal 
analysis  results,  training  point  descriptions,  or 
additional  specific  comments  drafted  by 
instructors. 

•  Provide  controls  to  show  or  hide  annotations 
or  to  present  operationally  significant  graphics 
overlays  on  the  2D  or  3D  view  of  the 
battlefield  during  playback,  including 
annotations  attached  to  single  entities. 

CHALLENGES  AND  RELATED  WORK 

There  are  numerous  playback  oriented  after  action 
review  tools  that  establish  the  precedents  for  solving 
fundamental  challenges  of  logging  exercise  events  and 
delivering  playback  at  distributed  stations  for  the 
training  audience.  The  AAIRS  debrief  toolset  attempts 
to  build  on  previous  efforts  by  constructing  an 
information-rich  timeline  which  provides  visual 
representation  of  key  events  in  an  exercise.  Further, 
the  visual  symbology  on  the  debrief  timelines  reflects 
the  results  of  automated  analyses  on  exercise  events 
and  EXFOR  actions  during  execution,  for  the  purpose 
of  offloading  some  of  the  analytical  workload  on 
human  instructors.  If  a  certain  proportion  of  possible 
training  points  are  automatically  detected  by  the  system 
and  offered  as  cues  or  bookmarks  for  the  after  action 
playback,  then  the  instructor  can  focus  on  catching 
non-standard  training  points,  or  making  a 
determination  of  which  training  points  to  emphasize  in 
debrief. 

Existing  AAR  playback  systems  have  been  built 
around  both  military  simulations  and  game  engines  and 
in  some  cases  a  mixture  of  the  two.  (Korris,  2004) 
describes  a  playback  AAR  capability  built  around  the 
Full  Spectrum  Warrior  game  for  the  Xbox  gaming 
console  as  a  squad  level  tactical  trainer.  This  provides 


a  visual  playback  with  a  time  slider  and  “VCR-like” 
controls. 

The  Aviation  Combined  Arms  Tactical  Trainer  - 
Aviation  Reconfigurable  Manned  Simulator 
(AVCATT-A)  provides  an  AAR  capability  based  on 
playback  of  logged  event  data  from  the  OneSAF 
Testbed  Baseline  (OTB),  as  well  as  recorded  voice  and 
digital  messages  (Knight  et  al,  2001).  AVCATT-A 
playback  capabilities  include  “jump-to”  features  and 
faster  than  real-time  stealth  view  features. 

A  distributed  AAR  system  was  developed  for  the  Joint 
Training  Experimentation  Program  (JTEP),  providing 
playback  with  2D  and  3D  display  capabilities  at  three 
distributed  sites  (Ford  et  al,  2004).  Due  to  limited 
bandwidth,  local  instances  of  playback  data,  consisting 
of  simulation  state  messages  and  recorded  voice  data, 
are  stored  at  each  site,  while  the  control  information 
regulating  the  playback  is  sent  over  the  network  in 
after  action. 

The  Dismounted  Infantry  Virtual  After  Action  Review 
System  (DIVAARS)  provides  stealth  capabilities  for 
instructors  to  observe  a  virtual  dismounted  exercise 
and  deliver  AAR  with  playback  (Clark  &  Lampton, 
2004).  The  AAR  tools  include  a  variety  of  controls  for 
manipulating  the  playback  and  incorporating  additional 
information  such  as  radio  communications.  DIVAARS 
also  uses  a  timeline  with  the  playback  capabilities, 
primarily  as  a  navigational  tool  as  opposed  to  a 
representational  method  for  presenting  exercise  events 
through  symbology. 

A  logical  next  step  that  builds  upon  frameworks 
providing  debrief  playback  capabilities  is  to 
incorporate  an  automated  processing  function  for 
highlighting  key  events  from  the  exercise.  The  Virtual 
Soldier  Skills  Assessment  (ViSSA)  prototype 
incorporates  an  analytical  capability  to  support  its 
AAR  functions,  in  a  training  domain  focused  on  small 
unit  dismounted  operations  (Gately  et  al,  2005). 
ViSSA  playback  tools  are  supplemented  by  a  visual 
timeline  where  significant  events  are  identified  for  the 
reviewer.  Similar  to  other  AAR  systems,  ViSSA 
supports  playback  of  both  visual  and  audio  data, 
however  the  analytical  component  does  not  attempt  to 
process  radio  communications  or  convey  their  content 
to  instructors.  This  kind  of  automated  causal  analysis 
is  one  of  the  key  objectives  in  the  AAIRS  debrief  tools. 

In  contrast  to  a  number  of  training  efforts  constructed 
such  that  smart  semi-automated  forces  can  “explain” 
their  actions  in  after  action  (Lane  et  al,  2005),  a 
common  objective  in  many  team  training  domains  is  to 


2006  Paper  No.  2962  Page  4  of  10 


Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2006 


derive  causal  explanations  for  the  actions  of  humans  in 
the  training  audience.  Due  to  the  complexity  of  this 
task,  particularly  in  large  scale  team  training,  the 
traditional  approach  is  to  rely  on  human  instructors  to 
perform  this  kind  of  diagnosis,  albeit  aided  by  stealth 
monitoring  capabilities.  However,  supporting 
automated  causal  analysis  requires  significantly  more 
depth  in  data  collection  than  a  log  of  simulation  events. 

The  task  of  building  utilities  for  adequate  data 
collection  in  support  of  after  action  review  is  often  one 
of  the  major  development  challenges  in  team  training 
applications.  For  some  domains  the  training 
requirements  are  limited  to  the  tabulation  of  statistical 
measures  of  performance  outcomes,  or  quantitative 
values  such  as  amount  of  fuel  or  munitions  expended, 
which  can  be  assembled  by  monitoring  one  stream  of 
data  direct  from  a  simulation  environment.  However, 
when  the  goal  is  to  provide  training  on  decision 
making  in  addition  to  outcomes,  and  in  fact  to  draw 
connections  between  the  two,  the  data  collection  task  is 
broader.  In  such  cases,  it  becomes  necessary  to  collect 
and  analyze  data  from  all  possible  input  sources, 
starting  from  the  interfaces  that  EXFOR  use  to  directly 
interact  with  each  other  and  with  the  training  system, 
and  ending  with  the  event  data  within  the  simulation. 

Particularly  in  team  operational  domains  like  combined 
arms,  the  executed  events  in  the  simulation  are  only 
half  the  story  of  what  happened.  As  an  indicator  of 
decision  making,  voice  communications  on  virtual 
radios  during  an  exercise  comprise  one  of  the  most 
valuable  sources  of  data,  and  therefore  a  ripe  source  to 
mine.  Language  processing  on  voice  communications 
as  they  happen  can  provide  direct  insight  into  the 
cognitive  states  of  the  EXFOR  participants  (Jensen  et 
al,  2005).  For  example,  determining  whether  a  given 
fire  mission  was  approved,  and  by  whom,  is  critical  for 
assessing  the  causes  if  the  fire  mission  led  to  a  conflict 
with  friendly  forces.  This  establishes  the  difference 
between  a  poor  operational  picture  and  mistaken 
execution. 

Although  language  processing  methods  cannot  achieve 
100%  recognition  accuracy  on  loosely  constrained 
speech,  they  can  provide  sufficient  accuracy  to 
generate  cues  for  the  instructor  reviewing  an  exercise. 
Such  cues  can  guide  an  instructor  not  only  where  to 
look  for  causes,  but  also  the  likely  conclusions  about 
what  the  causes  were,  without  requiring  linear 
playback  of  communications  audio. 

An  additional  challenge  for  an  AAR  system  that  will 
present  causal  analysis  results  is  the  means  of 
presentation.  The  objective  of  the  AAIRS  system  is  to 


provide  tools  not  only  for  preparing  and  controlling 
playback,  but  also  to  offer  the  timelines  themselves  as 
a  means  for  visually  representing  “what  happened.” 
Effectively,  this  can  establish  a  mutually  reinforcing 
relationship  between  debrief  playback  and  an  event 
timeline,  which  combine  to  make  trainee  understanding 
even  richer  than  when  either  is  used  in  isolation.  (Bahr 
et  al,  2005)  discuss  the  use  of  Graphic  User  Interface 
Embedded  Timelines  (GETs)  to  facilitate  rapid 
understanding  of  after  action  training  points.  The  GET 
tools  enumerate  a  range  of  functionalities  from 
snapshots  to  playback  to  “living  timelines”  with  event 
markers  and  duration  bars.  In  concept,  this  is  similar 
to  the  aims  of  the  implementation  of  the  AAIRS 
debrief  tools. 


SYSTEM  DESCRIPTION 

The  debrief  toolset  is  organized  around  the  concept  of 
individual  training  points.  A  training  point  is  any 
training  or  AAR  topic  to  be  presented  to  EXFOR  in  the 
debrief,  whether  automatically  generated  from  events 
detected  and  analyzed  by  AAIRS  during  execution,  or 
custom  created  by  an  instructor.  The  debrief  toolset 
provides  a  variety  of  utilities  for  tasks  such  as  indexing 
and  filtering  training  points,  and  creating  custom 
training  points,  but  the  focus  of  this  paper  is  the  visual 
timelines  and  the  debrief  playback  functions  of  the 
toolset. 

Debrief  Construction  Tasks 

The  Debrief  Construction  tool  interface  aims  to 
provide  a  quick  visual  reference  for  information 
relevant  to  the  current  training  point,  organized  along  a 
common  timeline.  By  representing  radio 

communications  and  simulation  events  visually  along  a 
timeline,  the  goal  is  to  provide  insight  as  to  the 
relationship  between  the  two,  especially  with  respect  to 
the  causes  of  errors  that  led  to  the  given  training  point. 

Reviewing  radio  communications 

The  communications  timelines  depict  radio 
communications  relevant  to  the  current  training  point. 
Communications  timelines  are  structured  around 
individual  radio  nets,  with  rows  for  each  active  listener 
on  the  net.  Individual  transmissions  are  represented 
with  the  times  and  durations  where  they  occurred,  and 
these  are  grouped  together  within  bounding  boxes  that 
represent  dialogs  between  two  or  more  speakers. 
Automated  language  processing  takes  place  during  the 
exercise  as  communications  are  sent,  to  detect  key 
content  that  reflects  EXFOR  decision  making  and 
commands.  In  cases  where  key  spoken  content  is 
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detected,  the  visual  representations  of  the  dialogs  are 
annotated  accordingly.  In  addition  to  serving  as  visual 
cues,  the  transmissions  are  playable  in  the  timeline 
interface,  so  that  the  audio  can  be  previewed  in  the 
process  of  reviewing  a  training  point.  Transmissions 


can  be  manually  annotated  if  desired.  Figure  1  below 
shows  the  communications  timelines  in  the  overall 
interface  for  the  initial  review  of  a  training  point  in  the 
Debrief  Construction  tool. 
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Figure  1.  Debrief  Construction  -  Timelines  with  Training  Point  Details 


Reviewing  causal  factors 

There  is  a  direct  relationship  between  the  causal  factors 
for  the  training  point  and  the  conclusions  about  the 
content  of  the  communications.  For  example,  in  the 
training  point  shown  in  Figure  1,  the  first  causal  factor 
identified  is  that  the  Artillery  fire  mission  was 
approved.  This  is  a  direct  result  of  the  automated 
detection  of  the  approval  dialog  on  the  ArCOFA  radio 
net  prior  to  the  execution  of  the  mission  (ArCOFA  is 
the  communications  plan  designated  name  for  the 
Artillery  Conduct  of  Fire  radio  net).  If  approval  had 
been  given  with  non-standard  language,  then  a 
secondary  route  to  the  same  conclusion  could  be 
carried  out  manually.  If  the  instructor  listens  to  the 
dialog  on  the  ArCOFA  net  and  hears  something  that 
amounts  to  approval  in  a  non-doctrinal  syntax  (e.g., 
“that  mission’s  good  to  go”),  then  with  a  manual 
annotation  of  the  transmission  as  an  approval,  the 
causal  analysis  results  are  updated  automatically. 


This  can  be  an  important  step  in  preparing  a  training 
point  for  debrief,  because  the  causal  factors  can  be 
displayed  in  bullet  form  during  debrief  presentation,  as 
a  slide-like  overlay  on  the  playback  view. 

Reviewing  simulation  events 

The  simulation  event  timeline  provides  a  reference  for 
the  tactical  situation  through  the  time  period  relevant  to 
the  current  training  point  (shown  at  the  bottom  of  the 
screen  in  Figure  1).  The  objective  is  to  show  the 
relative  times  for  when  movements  and  fire  missions 
took  place,  and  especially  highlight  critical  times 
associated  with  the  current  training  point,  such  as  a 
battlespace  geometry  conflict,  or  the  risk  of  a  conflict. 
The  content  of  this  timeline  is  specifically  designed  to 
visually  correspond  to  the  representations  in  a  fire 
series  worksheet,  using  symbology  that  will  be  familiar 
to  the  training  audience. 

In  the  example  shown,  the  training  point  involves  an 
Artillery  fire  mission  conflict  with  an  Air  section 
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executing  a  close  air  support  mission,  on  a  route  that 
passes  through  the  Artillery  trajectory  on  final 
approach.  To  represent  the  interval  in  which  the 
conflict  condition  occurred,  the  rows  for  the  Artillery 
unit  and  the  Air  unit  are  linked.  There  are  two  aircraft 
in  the  mission,  so  there  are  two  linked  intervals  shown, 
both  with  very  short  duration  due  to  the  high  speed  of 
the  aircraft  passing  through  the  trajectory. 

Customizing  playback  contents 

Part  of  the  purpose  for  structuring  the  debrief  tools 
around  individual  training  points  rather  than  one 
inclusive  playback  for  the  entire  exercise  is  to  facilitate 
the  process  of  preparing  targeted  after  action  review. 
Particularly  in  team  training  domains,  the  length  and 
complexity  of  the  training  exercises  means  that  a 
significant  burden  is  placed  on  instructors  if  they  are 
responsible  for  placing  all  markers  for  significant 
events  and  driving  the  debrief  playback  accordingly. 
Further,  in  a  complex  operational  exercise  (including 
distributed  and  live,  virtual  and  constructive  training), 
different  events  corresponding  to  unrelated  training 
concepts  may  occur  simultaneously  at  different 


locations,  for  which  different  viewing  positions  are 
best  suited  for  playback.  The  goal  is  to  reduce  this 
burden  by  automatically  constructing  a  default 
playback  vignette  for  each  training  point,  which  can  be 
customized  in  the  debrief  construction  process. 

The  playback  timeline  (just  above  the  simulation  event 
timeline  in  Figure  1)  represents  the  content  of  the 
debrief  playback,  including  markers  for  vignette  start 
times,  and  icons  for  the  audio  that  will  be  included 
from  recorded  radio  communications.  AAIRS 
automatically  chooses  an  initial  set  of  relevant 
communications  to  include  in  the  playback.  The 
playback  timeline  serves  both  to  show  what  is 
currently  included  in  the  planned  playback,  and  also  to 
allow  for  specific  transmissions  to  be  added  or 
removed.  AAIRS  also  automatically  configures  the 
viewing  position  and  angle  for  the  3D  view  in  the 
playback,  based  on  the  nature  of  the  training  point. 
Typically  a  final  step  in  preparing  the  debrief  for  a 
training  point  will  be  a  preview  of  the  playback. 
Figure  2  shows  the  debrief  tools  in  preview  mode. 
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Figure  2.  Debrief  Construction  -  Playback  Preview  with  Event  Timeline 


The  preview  mode  provides  a  means  to  see  in  advance 
what  the  playback  for  the  current  training  point  will 


look  like  when  presented  in  the  debrief.  The  playback 
timeline  and  the  simulation  event  timeline  persist  in 
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preview  mode,  with  a  time  slider  appearing  on  top  such 
that  the  3D  view  depicts  the  state  of  the  battlefield  at 
the  time  indicated.  The  playback  controls  include 
VCR-like  play,  pause,  and  skip-ahead  buttons,  as  well 
as  a  time  slider  that  can  be  freely  moved  directly  to  any 
time  of  interest.  In  preview  mode,  this  allows 
instructors  to  see  how  simulation  events  in  the  timeline 
are  correlated  with  events  in  the  3D  view.  For 
example,  in  Figure  2  above,  the  instructor  can  clearly 
see  that  the  “stay  above”  parameter  of  2000ft  above 
ground  level,  as  passed  to  the  aircraft,  was  not  high 
enough  to  clear  the  danger  area  for  the  Artillery 
trajectory.  A  2D  plan  view  can  also  be  turned  on, 
mapped  either  onto  graphic  terrain  or  a  map 
background. 

Navigational  tools  within  the  3D  viewer  are  available 
as  well,  to  change  the  position  or  viewing  angle  from 
the  default  values  initially  calculated  by  the  system, 
and  save  these  settings  with  the  training  point  for 
playback. 

Debrief  Presentation  Tasks 

After  completing  the  debrief  construction  process,  the 
task  of  presenting  the  debrief  in  after  action  is 
controlled  in  an  interface  almost  identical  to  the 
preview  mode.  As  discussed  earlier,  this  is  intentional 
in  order  to  provide  continuity  between  the  construction 
and  presentation  tasks,  as  they  are  frequently 
performed  by  the  same  individuals,  even  if  carried  out 
at  different  times  and  even  different  stations. 

The  debrief  presentation  interface  adds  controls  for 
navigating  between  training  points  in  the  debrief.  A 
simple  toolbar  provides  instructors  the  ability  to  show 
overlays  with  text  content  including  the  automatically 
generated  training  point  description  or  causal  factors, 
as  well  as  any  comment  text  they  created  in  reviewing 
the  exercise  training  points.  Additional  overlay  tools 
allow  for  the  display  of  graphics  and  control  measures 
representing  routes,  danger  areas,  and  other  significant 
visual  artifacts  in  the  battlefield.  This  is  particularly 
useful  for  training  points  that  involve  battlespace 
geometry  conflicts,  where  instructors  can  clearly  show 
actual  danger  areas  in  the  3D  space  and  allow  the 
exercising  force  to  see  where  intersections  occur.  Just 
as  in  preview  mode,  the  3D  view  can  be  freely 
navigated  to  change  the  point  of  view  during  playback. 

All  content  in  the  debrief,  including  any  custom 
training  points  or  custom  settings,  is  retained  with  data 
collected  during  the  exercise.  The  entire  database  can 
be  exported  as  a  take-home  product  from  training. 


USER  FEEDBACK 

The  evolution  of  the  AAIRS  debrief  toolset  involves  a 
spiral  process  of  user  feedback  through  workshops 
dating  from  the  initial  concept  stage  in  2003.  A 
preliminary  concept  of  operations  document  served  as 
the  first  concrete  vehicle  for  feedback  from  the  end 
user.  Initial  comments  emphasized  the  need  for 
supporting  both  2D  and  3D  views  of  the  battlefield  in 
playback.  Emphasis  was  also  given  to  intuitive 
placement  of  controls  in  the  user  interface  to  minimize 
the  need  for  “training  the  trainer”  and  frequent 
consultation  of  a  user  manual.  Early  user  feedback 
also  motivated  the  use  of  bullet-style  text  to  index  the 
causal  factors  identified  for  a  given  training  point. 
This  was  in  place  of  an  earlier  approach  which 
provided  causal  analysis  results  via  text  templates 
instantiated  with  data  specific  to  a  given  training  point. 
Clear,  concise  causal  factor  text  such  as  “Arty  mission 
was  approved”  provides  a  much  more  direct  suggestion 
of  the  conditions  in  which  the  training  point  occurred, 
as  well  as  where  to  look  for  more  information. 

After  the  concept  of  operations  document,  the  first 
implementation  of  the  debrief  toolset  was  presented  at 
a  subsequent  user  workshop  with  visiting  Marines  at 
the  CACCTUS  development  lab  in  Orlando.  This 
elicited  a  wealth  of  helpful  feedback,  including  the 
following: 

•  Marines  consistently  expressed  a  specific 
intention  to  be  highly  selective  about  the 
training  points  they  include  in  debrief.  Given 
the  time  constraints  of  the  training  setting, 
instructors  frequently  use  their  discretion  to 
select  the  most  significant  events  to  highlight 
in  after  action.  As  a  takeaway  for  the  debrief 
toolset,  this  amounts  to  a  requirement  to  allow 
the  assembly  of  a  debrief  that  supports  use 
cases  involving  a  small  number  of  detailed 
training  points  on  a  regular  basis,  while 
ensuring  that  the  full  set  of  automatically 
generated  training  points  can  be  assembled 
when  required  to  support  large  scale  training 
events. 

•  Marines  would  like  to  see  an  ability  to 
combine  training  points  when  they  involve 
related  simulation  events,  such  as 
simultaneous  fires  on  the  same  target. 

•  In  addition  to  the  automatic  analysis 
capabilities  which  find  errors  and  trace  causal 
factors,  instructors  would  like  the  ability  to 
index  through  a  list  of  all  fire  events  from  an 


2006  Paper  No.  2962  Page  8  of  10 


Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2006 


exercise,  regardless  of  outcome.  An  ability  to 
create  a  custom  training  point  for  any  fire 
event,  review  it  on  the  timelines,  and  rapidly 
generate  a  debrief  playback  accordingly,  can 
be  useful  for  positive  training  and  for  meeting 
other  flexible  training  goals. 

•  If  the  EXFOR  avoids  a  conflict  by  accident  in 
the  simulated  battlefield,  even  though  their 
decision-making  would  have  allowed  the 
conflict,  then  this  should  be  presentable  in  the 
debrief.  For  example,  if  a  mortar  fire  mission 
is  approved  for  a  certain  time  interval,  and  an 
aircraft  flies  through  the  space  of  the  mortar 
trajectory  during  that  interval,  but 
inadvertently  at  a  precise  time  after  a 
detonation  and  before  a  subsequent  fire  event 
in  the  simulation,  then  the  analysis  capabilities 
should  provide  the  ability  to  show  this 
decision-making  error  as  well  as  the  fortunate 
simulation  outcome. 

•  Annotations  on  entities  during  playback,  such 
as  an  altitude  tag  on  aircraft  icons,  provides 
helpful  information  for  debrief. 

•  Although  the  initial  default  is  for  AAIRS  to 
automatically  generate  the  point  of  view 
settings  for  3D  playback,  it  may  be  helpful  to 
expand  this  capability  to  generate  2  or  3 
camera  positions  for  each  training  point. 
During  playback  the  instructor  could  switch 
between  points  of  view  to  focus  on  different 
roles  or  battlefield  agents. 

•  A  grid  overlay  on  the  2D  view  in  either 
preview  or  playback  is  necessary  for  the 
training  audience  to  be  able  to  compare  what 
they  see  with  their  maps  and  also  get  a  sense 
of  scale. 

•  There  is  value  in  having  fire  series  names  and 
target  names  entered  into  the  system  during 
the  exercise,  so  that  these  can  be  referenced  in 
the  debrief.  This  translates  to  a  data 
collection  requirement. 

•  AAIRS  pre-selects  the  radio  nets  that  are 
relevant  to  a  given  training  point,  and 
provides  these  in  the  timeline  interface  for 
visual  and  audio  review.  Although  this 
frequently  captures  all  relevant 
communications,  instructors  still  want  the 
flexibility  to  choose  additional  radio  nets  to 
review  in  the  timeline  interface,  to  get  further 


context  for  the  events  surrounding  a  training 
point.  Similarly,  if  an  additional  net  is  added 
to  the  timeline  view,  the  debrief  construction 
interface  must  support  the  inclusion  of  any 
content  from  that  net  into  the  playback  for  the 
training  point  in  debrief  presentation. 

•  Chat  messaging  in  C4I  tools  is  increasingly 
being  used  as  an  operational  surrogate  for 
radio  communications,  and  therefore 
represents  an  important  future  additional 
source  of  data  reflecting  EXFOR  decision 
making. 

Many  of  the  general  comments  from  the  user 
community  at  workshops  have  been  consistent  with 
initial  requirements  and  objectives  established  early  in 
the  program.  This  is  a  positive  result,  as  it  essentially 
validates  the  vision  of  program  management,  as  well  as 
the  early  knowledge  engineering,  which  preceded 
design  and  development. 

Another  iteration  of  user  feedback  came  from  a 
subsequent  on-site  demonstration  of  the  system  at  the 
Twentynine  Palms  training  site.  Aside  from  generally 
positive  comments  about  the  toolset  as  a  whole,  further 
specific  input  was  noted,  including: 

•  It  is  important  for  the  modeling  of  indirect  fire 
trajectory  arcs  and  time  of  flight  to  match  the 
tables  that  Marines  use  in  their  own  decision 
making.  This  means  that  the  simulation 
models  need  to  be  consistent  with  the 
analytical  models  used  by  AAIRS,  which 
need  to  be  consistent  with  the  EXFOR  tools. 

•  Due  to  the  potentially  large  size  of  training 
events  to  be  held  with  the  system,  there  can  be 
many  communications  taking  place 
simultaneously  on  a  given  radio  net.  This 
suggests  an  additional  goal  for  the 
representations  in  the  communications 
timelines,  such  that  the  reviewer  can  be  given 
visual  cues  showing  which  communications 
specifically  do  apply  to  the  current  training 
point. 

•  Reiterated  interest  in  capabilities  to  create 
training  points  for  any  exercise  events,  for  any 
reason  at  the  instructor’s  discretion.  An  initial 
implementation  of  this  has  since  been 
completed. 
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THE  WAY  FORWARD 

There  are  both  near  term  and  long  term  plans  for  the 
continued  development  and  deployment  of  the  AAIRS 
debrief  toolset.  With  the  first  installed  versions  of  the 
toolset  at  CAST  sites,  the  user  community  will  be 
generating  feedback  and  specific  requests,  which  will 
be  supported  on  the  development  side  to  tailor  the 
toolset  to  their  needs  as  familiarity  increases.  Longer 
term  plans  for  the  toolset  include: 

•  The  export  capability  on  the  training  database 
will  be  extended  to  support  playback  on 
machines  where  a  full  installation  of  the 
CACCTUS  baseline  is  not  available. 

•  Ongoing  advances  in  speech  recognition 
technology  will  be  incorporated  into  the 
toolset’s  analysis  component  for  processing 
voice  communications. 

•  Additional  sources  of  data,  including  a  variety 

of  C4I  tools  that  EXFOR  may  bring  to 
training  exercises,  will  be  supported  in  the 
system  by  way  of  data  collection  in  the 
exercise  database.  Additional  data  sources 
will  also  provide  further  insight  into  cognitive 
states  and  models  held  by  the  EXFOR.  For 
example,  if  they  use  digital  tools  to  maintain 
operational  maps,  then  these  provide  insight 
into  “perceived  truth,”  which  can  be 

contrasted  with  ground  truth  as  reflected  by 
simulation  data,  and  shown  in  playback. 

•  Program  goals  include  support  for 

simultaneous  distributed  exercises  run  with 
participation  from  different  training  sites  at 
different  geographic  locations.  The 

architecture  for  data  collection  and  debrief 
construction  and  delivery  will  be  extended  to 
accommodate  this  training  goal. 

•  Program  goals  also  include  support  for  live, 

virtual  and  constructive  training,  where 

simulated  elements  may  be  included  with  live 
elements  in  the  same  exercise.  This  will  also 
require  extensions  to  the  data  collection  and 
debrief  schemes  in  the  toolset. 


Cole  Engineering  Services,  Inc.  Additional 
collaboration  with  the  NAVAIR  Orlando  Concept 
Development  and  Integration  Lab  facilitated  the 
integration  of  debrief  capabilities  with  the  CACCTUS 
architecture. 
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